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METHOD AND SYSTEM FOR CORRELATING DATA STREAMS 



BACKGROUND OF THE INVENTION 

It is often quite useful to store the contents of a 
data stream for later retrieval. A well-known example is 
5 streaming audio data that is stored to a recording medium 
for playback at a later time. 

Some situations require that multiple related data 
streams be recorded simultaneously but in separate data 
files. If the data streams are related, it is often 
10 desirable that the data streams be correlated. That is, 

given a position or data point in the first data stream, it 
is often useful to know the corresponding location of 
related data in the second data stream. 

One application potentially requiring correlated data 
15 streams involves the recording of certain audio-video data. 
Generally, the contents of a video data stream must be 
correlated to an audio data stream. When these data 
streams are correlated as mentioned, it is possible to 
playback a selected section of video data along with the 
2 0 corresponding audio portion of such a recording. 

An example of an application that correlates streams 
of events is SYSLOGD, which is a UNIX utility application 
that logs events from multiple sources in a central file 
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based on t ime- stamps . According to the principles of 
SYSLOGD, messages received from multiple event streams are 
stored with corresponding time-stamps in a central file 
system. 

5 In the SYSLOGD application, storage of event records, 

each consisting of an event such as a received message and 
a corresponding time-stamp, is typically sufficient to 
establish a relationship between events occurring on two or 
more distributed computer systems. For example, a message 
10 received in one event stream can be correlated to an event 
in another event stream by traversing a list of time-stamps 
and retrieving messages having time-stamps of nearly the 
same time. 

While event logging systems such as SYSLOGD are useful 
15 for tasks such as computer system administration, they are 
generally not adequate for correlating streams of data 
including audio or video frames. 

SUMMARY OF THE INVENTION 

The present invention is a novel apparatus and method 

20 generally directed towards correlating data among multiple 
data streams based on a use of time-stamps and related 
positional information. One potential application is a 
closed captioned system in which text is generated for 
display along with corresponding video data. For example, 

25 closed captions are textual transcriptions of an audio 
track of a television show that are displayed on a 
television screen at approximately the same time as 
corresponding video data. According to the principles of 
the present invention, the audio track or audio data stream 
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can be correlated with the textual data stream to support 
synchronization of such streams during playback. 

More particularly, one aspect of the present invention 
involves correlating time-sequenced data streams by 
5 identifying certain events within a first data stream and 
generating positional information indicating where data 
associated with the events are located in a corresponding 
storage medium such as a digital file or where the data is 
located within the data stream. Each detected event is 

10 preferably assigned a time-stamp reflecting a time when it 
was detected based on a system clock. The time-stamp 
serves as an index or reference in which events are 
correlated between related data streams. For example, 
events in different data streams having nearly common time- 

15 stamps indicate that such events were detected at nearly 

the same time and that data associated with such events are 
therefore related . 

In one embodiment, a word processing data stream is 
correlated to a corresponding audio data stream. The time- 

2 0 stamps are potentially generated by a common system logger 

that time-stamps events detected in the second data stream. 

Another aspect of the present invention involves 
correlating events in a first and second data stream based 
on data pointers. This involves detecting events in the 
25 first and second data stream and assigning corresponding 
data pointers to the detected events. These data pointers 
indicate where data associated with detected events are 
stored in the first and second data stream. Time- stamps 
are assigned to data pointers associated with the detected 

3 0 events. These time-stamps indicate a time when 
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corresponding events were detected in a data stream. For 
each detected event in a data stream, a data set is 
maintained in an index file. Preferably, each data set 
includes a time-stamp indicating when an event was detected 
5 and a corresponding data pointer indicating a location 
where the data associated with the corresponding event is 
stored in the data stream or data file. 

To find a corresponding event in a correlated data 
stream, an event in the first data stream is identified and 

10 its corresponding time-stamp is determined. The time-stamp 
of the event in the first data stream is then compared to 
time-stamps of events occurring in the second data stream. 
In this way, the time- stamps are used as an index to locate 
where the data associated with the corresponding event is 

15 stored in the second data stream. For example, a time- 
stamp of an event in the second data stream nearest the 
time-stamp of the identified event in the first stream is 
determined and the data pointer associated with the second 
time-stamp is then used to locate the event-associated data 

2 0 in the second data stream that correlates with event - 

associated data in the first data stream. 

In one embodiment, time -stamps assigned to events in 
the first and second data stream are generated from a 
common system clock. Alternatively, time-stamps are 
25 generated by separately located system loggers having 
synchronized system clocks. 

In certain applications, at least one data stream of 
the correlated data streams is generated or presented 
asynchronously. For example, in one application, the 

3 0 playback rate of an audio recording is varied resulting in 
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an asynchronous data presentation. In a similar but more 
specific application such as the transcription of 
prerecorded speech, the playback rate of an audio file is 
optionally adjusted by a control unit that automatically 
5 changes the speed of the audio playback to match a 

corresponding text entry rate. The text data stream is 
linked to the speech in the original audio recording even 
though the audio recording is played back at a time-varying 
rate while the text was entered. 

10 Another aspect of the present invention involves 

correlating more than two data streams and storing related 
time-stamped event data pointers to a storage device for 
later retrieval. As previously mentioned, the time-stamps 
indicate a reference time when a particular event was 

15 detected while a corresponding data pointer indicates a 
location where data associated with that event was stored 
in the data file. 

Events in a data stream are optionally defined in many 
ways. For example, an event may be based on a passage of a 

20 predefined interval of time. Alternatively, an event is 
optionally defined as a random or pseudo-random event such 
as the occurrence of a particular word or character in a 
data stream. 

The present invention has many advantages over the 
25 prior art. For example, it is possible to correlate data 
streams and events in separate data files without requiring 
an exhaustive search to identify related data in either of 
the streams. Positional information indicates a location 
where data associated with an event is stored in a data 
30 stream or data file, while a time-stamp indicates a time 
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when an event was detected. Since time- stamps are 
generated from a common system clock, event-associated data 
from a first data stream are correlated with event- 
associated data in a second data stream based on time. A 
5 search for related data is therefore a simple task because 
the time-stamps serve as an easy-to-use reference index. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing and other objects, features and 

advantages of the invention will be apparent from the 
10 following more particular description of preferred 

embodiments of the invention, as illustrated in the 

accompanying drawings in which like reference characters 

refer to the same parts throughout the different views. 

The drawings are not necessarily to scale, emphasis instead 
15 being placed upon illustrating the principles of the 

invention . 

Fig. 1 is a block diagram of a recording system for 
correlating data streams including a search interface 
according to the principles of the present invention. 
2 0 Fig. 2 is a table of sample events in a first and 

second data stream including data pointer information and 
corresponding time-stamps according to the principles of 
the present invention. 

Fig. 3 is a graph illustrating sample events as 
25 defined in Fig. 2 according to the principles of the 
present invention , 

Fig. 4 is a recording system that correlates data 
streams from multiple sources interconnected via a 
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communication link according to the principles of the 
present invention , 

Fig. 5 is a recording system utilizing a rate 
controller to automatically control the speed of an audio 
5 data stream so that it is matches a streaming rate of a 
corresponding text-based input according to the principles 
of the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

A description of preferred embodiments of the 

10 invention follows. 

The principles of the present invention can be used to 
establish a correspondence between data streams by events 
correlated in time. One example of a "data stream" is a 
sequence of video and audio frames read from a media file 

15 that can be played back through an application such as 

Microsoft Windows Media Player. A second example of a data 
stream is a series of words produced by a word processing 
application. When two data streams have been placed in 
correspondence, or correlated/ each item or group of items 

20 in a first data stream is associated with an item or group 
of items in a second data stream. 

For each data stream, a set of "events" is chosen so 
that the data stream can be partitioned into intervals that 
are meaningful with respect to the correlation task. For 

25 example, in a video player application, events can 

correspond to a change in the playback speed and direction, 
such as fast forward, reverse, play and stop. These are 
relevant events because they alter the speed and direction 
of data transmitted to a display device. In a word 
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processing application, events can correspond to editing 
operations performed on a document, such as the insertion 
or deletion of words. 

Associated with each event is a "data pointer" that 
5 indexes into the data stream at the point where the event 
occurs. In the video player application, the data pointer 
can be the frame number for a particular video frame at 
which there was a change in the playback rate. Likewise, in 
a word processing application, the data pointer can be a 

10 global index into a list of words, indicating the point at 
which an insertion or deletion operation occurs. 

Fig. 1 is a block diagram illustrating various 
components of the present invention for automatically 
correlating multiple streams of data. Generally, data 

15 stream correlator system 200 supports indexing data items 
in a first data stream 205 to data items in a second data 
stream 2 55 using time of detection or recording as a 
reference. Although Fig. 1 illustrates how two data 
streams can be correlated, such an embodiment is optionally 

2 0 expanded to support correlation of more than two data 

streams . 

More specifically, associated with first data stream 
2 05 is an event detector module 215 and a data pointer 
generator module 220, The event detector 215 identifies 
25 significant events, while the data pointer generator 220 
associates each event to a location in the data stream 205 
or data storage 23 0. The output of the data pointer 
generator module 220 is a sequence of pairs consisting of 
an event, E , and an associated data pointer, P. The j fch 

3 0 event in a sequence generated from the ith data stream is 
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denoted E- • . Its associated data pointer is denoted P- • . 

The data pointer generator module 22 0 for the itii data 
stream produces a sequence of event -data pointer 

pairs: (^,o,/^.o)' (^u,^.2)' ••• 

5 Sequences of event -data pointer pairs form the inputs 

to event-data pointer logger 250. The logger assigns a 

unique time stamp T^j to each pair (E^^j.P^j) . The resulting 

event-data pointer record 152, (E.j,P.j,T-j), is then stored 

in a data stream index file #1 235. There is usually one 

10 index file for each data stream. 

In a similar manner, an event -data pointer record can 
be produced and stored in data stream index file #2 285 for 
second data stream 255. 

Comparison between index files forms the basis for 

15 data stream correlation. Events which are in close 

proximity in time based on the time-stamps in the index 
files establish a correspondence between their associated 
data items. For example, two data streams can include 
corresponding audio and video data. By comparing time- 

2 0 stamps in their associated event -data pointer records, it 
is known which audio samples correspond to a particular 
video frame and vice-versa. For example, a video frame of 
a weather-person and temperature map will have nearly the 
same time- stamp as a corresponding audio segment of the 

25 weather-person discussing matters such as temperatures on 
the temperature map. In other words, since such events or 
segments were logged at nearly the same time, they will 
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have nearly the same time-stamps associated with their 
stored data. 

Streams of data can either be produced or consumed, 
depending upon the application. For example, in a media 
5 player application, streams of video and audio data are 
consumed by the player, which passes the video frames to a 
display and the audio samples to a sound system. The video 
and audio data may originate, for example, from a data file 
stored on a hard disk or from packets transmitted over a 
10 network. 

In the exemplary application as shown in Fig. 1 we 
denote a system that produces a data stream as a "producer" 
module 210 and a system that consumes a data stream as a 
''consumer" module 260. The embodiment of either module can 

15 be a software program, a hardware device, or some 

combination of both. In the media player application, the 
consumer is a software program. Alternatively, a producer 
for audio and video data streams could be a VCR or similar 
hardware device. A module can also be a "'producer- 

2 0 consumer" that produces data items during certain times and 
consumes data items at other times. 

In Fig. 1, a producer module 210 generates a first 
data stream 2 05 of information such as words in a text 
processing application. As data from the first data stream 

25 205 is generated, contents of the first data stream 205 are 
stored in first data storage unit 230 via first storage 
management unit 225. 

Producer module 210 also generates a first control 
stream 206. The control stream contains information 

30 describing attributes of the associated data stream. For 



9^18 ,2 013-000 (PD99-2 759) 



example, in a text -processing application the data stream 
can consist of words and characters, while the control 
stream can specify the relative order of words in the 
document and whether words are being inserted, appended, or 
5 deleted. 

The first memory management unit 225 utilizes the 
control stream to determine how to enter the data items in 
the data stream into the data storage unit 230. In a text- 
processing application, for example, the position of a word 
10 data item including information about whether it is being 
inserted or appended can be used to place the word in the 
correct position in a list of words contained in the data 
storage unit . 

First event detector 215 analyzes the first control 
15 stream 206 and searches for specific events. An event can 
be defined in many ways. For example, an event can be 
defined as the occurrence of an insert, append, delete, or 
replace operation in the first control stream as a user 
generates textual data. 
20 As events within the first control stream 206 are 

detected by first event detector 215, a sequence of events 

217, J , is communicated to the first data pointer 

generator 220, The generator also receives positional 
information 218 from the first storage management unit 225 

2 5 such as a data pointer, P^j , indicating where the data 

associated with the event is stored in the data storage 
unit 230, First data pointer generator 220 combines the 
event and data pointer and outputs a sequence of event -data 
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pointer pairs 2 52, (E^j^P^j) , to event -data pointer logger 
250 . 

Note that in certain application of the present 
invention, a control stream can contain additional control 
5 information that is used by the event detector but not used 
by the storage management unit. In this instance, the 
storage management unit would ignore extraneous control 
information. 

In situations where the event detector does not 

10 require any additional control information and the control 
and data streams are accessible via an interface, it is 
possible to apply the principles of the present invention 
to an existing producer or consumer module without 
modifying its operation. 

15 For example, a standard commercial digital video tape 

player can be used in the video-audio correlation 
application without any modification. Such a video tape 
player can be interfaced to a computer using a standard 
IEEE 1394 (Firewire) interconnect. The player then 

2 0 communicates both the video and audio information and 
control information over the connection to indicate the 
current playback settings. 

Event-data logger 250 receives two sequences of event- 
data pointer pairs, 251 and 252. A time-stamp for each 

2 5 event -data pointer pair is generated based on a system 

clock. The time-stamp reflects the time at which the pair 
was received by the logger and is appended to the event- 
data pointer pair, resulting in an event -data pointer 

record 152, (E,^j,P.j,T.J. 
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Each event -data record includes the event E-j that was 

detected, a data pointer j indicating the location where 

the data associated with the event is stored in a 
corresponding memory device (or optionally a location where 
5 the event data occurred in the data stream) , and a time- 
stamp T- J , The event -data pointer records 152 resulting 

from the first data stream are stored in a first data 
stream index file 235. 

Fig. 1 illustrates a consumer module 2 60 that receives 

10 as input a second data stream 255. The consumer module 

generates a second control stream 2 56 that second storage 
management unit 275 uses to retrieve data items from second 
data storage unit 280. Note that in contrast to the first 
data stream 2 05, the second data stream 255 is produced 

15 based on data items 277 that are retrieved from data 

storage unit 280. Thus, a consumer module 260 reads items 
from pre-existing data storage while a producer module 
places items into storage. A producer-consumer module would 
exhibit both of these behaviors. 

2 0 In a similar manner as described above for the first 

data stream 2 05, a second data stream index 2 85 is 
generated for events detected in a second control stream 
256. Preferably, event-data pointer records 152 for the 

second data stream 255, (£"2,^ 5 Aj?^2,y ) ^ stored in second 

25 data stream index file 285. It should be noted that data 
sets 152 generated for each data stream are optionally 
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stored in a common index file rather than separate index 
files as shown in Fig, 1. 

The embodiment of the present invention depicted in 
Fig. 1 involves a producer module 210 and a consumer module 
5 260, In general, there is no requirement that producer and 
consumer modules occur in pairs. For example, an embodiment 
of the present invention can include any number of 
producers and consumers . 

Search requests 2 95 submitted through search interface 

10 290 are serviced based on stored data items and 

corresponding index information. A search request 
typically involves identifying a data pointer or event 
associated with one data stream and determining a 
corresponding data pointer or event in another related data 

15 steam. This will be discussed later in the specification. 

It should be noted that the first data stream 2 05 and 
second data stream 255 can be of different types including 
different types of data items. In one embodiment, a data 
stream is generated at a known rate where the received data 

20 is sequentially stored in a data file. One application 

where data is generated or presented at a known rate is the 
playback or actual generation of a real-time audio file. 

In alternate embodiments, however, data is generated 
or presented at a variable rate or in an asynchronous 

25 fashion unlike real-time audio data. In one situation, 

data items received towards the end of an asynchronous data 
stream correspond to data that is to be stored at the 
beginning or middle of a data file. For example, during an 
editing phase, textual data generated by a user at a 

3 0 keyboard can be stored any where in a corresponding word 
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processing file even though the data is generated later in 
time within a data stream. In other situations, 
transmission of the data may be asynchronous with respect 
to the rate at which the data is transmitted. For example, 
5 an audio stream recorded in real-time is optionally played 
back at variable rates in generating a data stream. Data 
stream correlator 200 is versatile as it can be used to 
correlate these and other types of data streams. 

Fig. 2 is a table of event data sets 152 for a first 

10 and second data stream according to the principles of 
correlating data as described for Fig. 1. As mentioned, 
certain events within each of the control streams are 
detected and stored along with corresponding data pointers 
and time-stamps. Events such as e^ correspond with 

15 detected events in the first control stream 206 while f^ 
corresponds with detected events in the second control 
stream 256. Positional information such as data pointers 
reference a location in a respective data file where data 
associated with an event is stored while time stamps 

2 0 reference a point in time when an event was logged. 

Fig. 3 is a graph of sample trigger events as recorded 
over time according to sample data in the table of Fig. 2. 
According to the principles of the present invention, 
events in one data file are indexed to events in a second 

2 5 data file based on time- stamps and corresponding data 

pointers . 

Consider event e3 in the first data file. To determine 
a corresponding event in the second data file associated 
with e3, the event data set 152 associated with the event 

3 0 is first located. It is known from such data that the 
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corresponding time-stamp for 63 is t5=33 . This time-stamp 
serves as an index into time-line of Fig, 3. 

In the course of determining a corresponding event in 
a second data stream, time-stamps associated with events in 
5 the second data file are compared with time-stamp tg=33. 
There are two events in second data file that have nearly 
the same time-stamps as our trigger event e3 in the first 
data stream: t5=32 or tj=AA. It is known based on closer 
proximity in time that the event in second data file 
10 related to t5=32 is the closest matching event in the 

second data stream. That is, 32 is closer to 33 than 44. 
Consequently, event e3 in the first data file corresponds 
with event f3 in the second data file. Location of event f3 
is determined using data pointer associated with time-stamp 

15 tg. 

In a similar manner, events from second data stream 
255 incorporated in second data file are indexed to events 
within first data stream 205 stored in the first data file. 
For example, consider event in second data file and 

20 related time-stamp data tg = 54 . Time-stamps in event data 
sets 152 of the first data file are compared to the time- 
stamp t9=54. Based on corresponding time-stamps, it is 
known that the two events closest to in the first data 
stream or file are and That is, event has a time 

25 stamp of t8=47 which is prior to time-stamp t9=54 

corresponding to f^. Event in the first data file has a 
time-stamp of t;^o=55, which was recorded after but at nearly 
the same time as event f^^ Hence, event e^ in the first 
data file corresponds with event in second data file. 
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Other cause-effect correlation options are also 
possible. For example, it is possible to link to the data 
pointer with the closest time stamp occurring before a 
given time stamp, or link the data pointer with the closest 
5 time stamp occurring after a given time stamp. 

It should be noted that the recorded data file and 
associated index information can be processed in different 
ways to locate an event in a data stream or corresponding 
data file. For example, given a particular system clock 

10 time, the event data sets 152 are optionally searched to 
determine a closest time-stamped data set. Based on a 
corresponding data pointer of a nearest time- stamped data 
set 152, the whereabouts of a corresponding event in a 
related data stream can be determined. 

15 Alternatively, given a position in the data stream or 

data file, a corresponding location in a correlated data 
stream can be determined by first identifying the closest 
data pointer in the index of the corresponding data stream. 
Thereafter, the information in the event data set 152 

2 0 corresponding location is determined by comparing time- 
stamps . 

This unique method of correlating data streams is 
advantageous over other methods. For example, according to 
the principles of the present invention, events are 
25 correlated with each other based on time-stamps indicating 
when an event was detected in a data stream and data 
pointers indicating a location in memory where an event is 
stored in the data file. It is often difficult to locate 
events stored in a data file when positional information 
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such as data pointers are not maintained for tracking 
corresponding events. 

Consider a case where only events transmitted to the 
logger 250 are time-stamped. In many applications, given a 
5 particular event, it is difficult to locate a corresponding 
data item because the time-stamp associated with an event 
may not linearly correspond to a position of the data item 
in the data file or data stream. That is, data from a data 
stream may not be received in a continuous linear manner 

10 over time. Rather, intermittent events within a data 

stream may be sporadically received and thereafter stored 
to a data file. This typically renders it a formidable 
task to locate a data item in a data stream or data file 
because a whole file must be searched to find a particular 

15 data item. 

Application to On-Line Note-Taking 

In one embodiment, the first data stream 205 is text 
data generated by a user in response to an audio-visual 
20 presentation. It is desirable to correlate a series of 

notes taken during the course of a lecture with an audio- 
visual recording of the lecture. This can be accomplished 
using the principles of the present invention. 

For example, the first data stream 205 is optionally 

2 5 text -based data generated by a user at a keyboard. The 

second data stream 2 55 is audio data corresponding to the 
lecturer's presentation. It is generated by a second 
producer module that represents the microphone system in 
the lecture hall. Note that in this instance of the present 

3 0 invention, there are two producer modules generating 
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streams of data in real-time. The audio and text-based data 
stream can be correlated via data stream correlator system 
200 as previously described except consumer module 260 is 
now a producer module also. 
5 An example of events for a text -based data stream are 

"append," ''insert/' and ''delete." "Append" events are 
generated when a word is added at the end of a document, 
while "insert" events are generated when a user inserts a 
word into the document near previously typed text . 

10 Finally, "delete" events are generated when a word is 
deleted from a document. 

As mentioned, when events are detected, a data pointer 
is generated indicating a position of the associated word 
within the corresponding data stream or document file where 

15 it is recorded. This process of creating data sets is 
advantageous because an index of information for 
correlating data streams is generated as data of a 
respective data stream is being stored to a data file. 
Hence, data items in one stream are correlated with data 

2 0 items in another stream while data in each stream is 

received and stored to memory. 

Data pointer values do not necessarily have to be 
actual physical addresses, instead these values can be 
arbitrarily specified, but linked to physical addresses via 
25 a lookup table or address map. 

In one application, values of the data pointers are 
spaced to accommodate insertions into previously entered 
text. Consider records including an event and 
corresponding data pointer indicating an index into a 

3 0 document file specifying where a word is stored in a 
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document . That is, in a record such as (append, 300,25), 
''append" is the type of event detected and ''3 00" is the 
index where the appended word is stored. The number "25" 
of the sample data set is a time-stamp indicating when the 
5 append event was logged based on a reference clock. For 
example, the append event occurred at time marker 25. 

If a sequence of events/data pointer pairs is 
(append, 300 , 25) , (append, 400, 30) , and (append, 500 , 40) , an 
event such as "insert" can be appropriately added at a 
10 later time. For instance, an "insert'' event such as 
(insert , 3 50, 50) can be added to a corresponding index 
without effecting the recorded event /data pointer pairs for 
already existing pairs (append, 300 , 25) and (append, 
400, 30) . 

15 In the previous example, the values of data pointers 

are averaged to generate an appropriate new data pointer 
for the event corresponding to an insertion into the 
document. For example, the data pointer for the previously 
discussed "insert" event is determined by averaging 300 and 

20 400 to produce a data pointer of 350. This technique is 
advantageous because data pointers for previously recorded 
events need not be changed when an event is inserted at a 
later time. 

Predefined events in the second data stream 255 of our 

2 5 exemplary embodiment are the recording of a fixed number of 

audio samples. For example, an event can be generated 
whenever one second of new audio had been recorded. Data 
pointers associated with the detected event preferably 
identify the relative or absolute position of the audio 

3 0 tape when the corresponding event was detected. In one 
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application, the data pointer is a burned- in- time- code of a 
tape, indicating the absolute position in the audio data 
stream where an event is detected. 

Optional features for correlating events according to 
5 the principles of the present invention include event 
filtering and data pointer interpolation. 

Event filtering involves consolidating events recorded 
for a correlated data stream. Consider the situation where 
an event such as ''insert" or ''append" is recorded based on 

10 user text input. At a later time, the word corresponding 
to the "append" or "insert" event may be consolidated in 
the index with a "delete" event, such that the "insert" and 
corresponding "delete" event are removed from the index. 
Consider a sample sequence of events such as 

15 (append, 500, 20) , (append, 600 , 25) , (append, 700 , 35) , 

(delete, 500, 40) , (insert, 650, 50) . After event filtering, 
events (append, 500 , 20) and (delete, 500 , 40) are consolidated 
and the sequence becomes: (append, 60 0 , 25) , (insert , 65 0 , 50) , 
(append, 700, 35) . 

20 Without event filtering, correlating a text data 

stream corresponding to the real-time creation of a 
document can be problematic. Any word which occurs in 
association with a certain time-stamped event could be 
removed by a later event. Event filtering consolidates 

2 5 these events and produces a set of records which describe 
the consolidated data stream. 

Less processing power is required to utilize such a 
newly consolidated index. Additionally, less memory is 
required to store the index. 
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Data pointers in the original list can occur out-of- 
order with respect to a time - stamp , For example, the delete 
and insert events above can occur out-of-order. Following 
event filtering, the list can be ordered with respect to 
5 data pointer, and there will be only one event associated 
with any particular data pointer. 

Data pointer interpolation is used to generate a 
regular stream of data pointers when a sparse set of events 
occurs in one of the control streams. For instance, 

10 consider two correlated streams of data where one stream 
has associated with it a sparse set of events. Without 
events and time-stamps in both streams, there are no events 
to precisely synchronize the streams of data with each 
other. According to this technique, events are added at 

15 regular intervals of time when there are few detected 
events in a particular data stream. This technique of 
adding events is often advantageous in situations where it 
is necessary to precisely synchronize related data streams. 
Fig. 4 is a block diagram of a note-taking system 

2 0 incorporating the principles of the present invention. 

Generally, multiple streams of data generated by portable 
computers are correlated with a corresponding audio-video 
data stream and slide presentation. This embodiment is 
particularly useful in lecture settings where students with 

2 5 portable computers take notes while the lecture is 

simultaneously recorded using video capture equipment. 
Following a lecture, students can compare notes taken at a 
particular time of the lecture based on the principles of 
the present invention. 
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A first computer 520 and second computer 540 are in 
communication with data processing unit 56 0 via 
communication link 510 such as a wireless or Internet 
connection. The first computer 52 0 supports a word 
5 processing application 522 enabling users such as students 
to take notes of a presentation such as a live lecture. A 
data stream and corresponding data file is created as a 
user at first computer 52 0 records notes in word processor 
application 522. Events within the data stream of notes 

10 generated by user are detected at event triggered data 

pointer generator 524, which produces a corresponding data 
pointer for each detected event , 

As mentioned earlier, the type of events that trigger 
the event triggered data pointer generator 524 are 

15 predefined for a particular application. Preferably, the 
class of trigger events is chosen so that a reasonable 
number of event data sets are generated for correlating 
respective data streams. As mentioned, trigger events can 
be inputs such as insertions, deletions and appending of 

2 0 text -based information. 

Second computer 540 provides similar functionality as 
that described in the first computer 520 for a second user 
taking notes. Notably, recording system 500 optionally 
supports other computers in addition to first computer 52 0 

2 5 and second computer 54 0 as shown. 

During a presentation of information at slide display 
device 580, an audio-video recording unit 585 captures 
details of, for example, a corresponding slide presentation 
including a lecturer delivering a speech. A data stream 

3 0 generated by the audio-video recording unit 585 is captured 
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for storage in a data file at data processing unit 560. A 
stream of data generated by the audio-video recording unit 
585 is monitored for events at event triggered data pointer 
generator 570. In a similar manner, slide display device 
5 580 generates a stream of data that is monitored at a 

second event triggered data pointer generator 564 in data 
processing unit 560. It should be noted that different 
events are detected for each data stream based on a 
corresponding control stream. 

10 As events are detected within a corresponding control 

stream, information regarding the occurrence of such events 
are stored in an index file by a corresponding event 
recording unit that generates event data sets 152. Since 
the time-stamp associated with each event serves as an 

15 index to correlate events in multiple streams, it is 
advantageous to synchronize the time-stamp generating 
system clocks among processing units via synchronizer 562 
to assure that the time-stamps generated at different 
locations are based on a common clock setting. Thus, as 

2 0 events occur in a corresponding data stream at remotely 
located devices, the events will be time-stamped relative 
to a virtually common system clock. 

Events detected at event detector 524 trigger event 
recording unit 526, which is a combination of Data Pointer 

25 Generator 220 and Event-Data Pointer Logger 250, to record 
an event data set 152 including event type, data pointer 
and corresponding time-stamp in memory storage unit 528. 
In a similar manner, events detected at event detector 544 
in second computer 540 are stored in memory storage unit 

30 548 via event recording unit 546. At data processing unit 
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560, events detected at event detector 564 and event 
detector 570 are stored in memory storage unit 568 via 
event recording unit 566. Again, all events and related 
information is stored with a corresponding time-stamp 
5 generated from respective and preferably synchronized 

system clocks. The index information created for each data 
stream such as data sets 152 can be used a at a later time 
to correlate data events among the data streams. For 
example, students attending the lecture can compare notes 
10 taken at a particular point in time according to the 
principles of the present invention. 

Application to the Generation of Closed Captions 

Another application of data stream correlator system 
2 00 is a system that permits a human operator to create 
15 closed captions for video or audio content. Closed 

captions are textual transcriptions of an audio track of a 
television show that are displayed on a television screen 
at approximately the same time as corresponding video 
frames , 

2 0 According to the principles of the present invention, 

the audio track or audio data stream can be correlated with 
the textual data stream to support synchronization of such 
streams during playback. The block diagram of Fig. 1 is 
appropriate for this application. In this case, the 

2 5 producer is a text -entry program that permits a human 

operator to type words corresponding to the audio track of 
a television program. The data stream 205 in this case will 
be a sequence of words. The associated control stream 2 06 
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would consist of a single token, called append, which will 
occur whenever a new word is generated. 

The event detector 215 in this application can simply 
count the number of words that are typed and generate a new 
5 event -data pointer record every time a certain number of 
words has been typed. For example, a record could be 
created whenever there was one new word, or whenever there 
were 10 new words. Increasing the word count reduces the 
number of records in the log, thereby reducing 

10 communication bandwidth and storage requirements However, 
this is done at the expense of accuracy in locating 
individual words in a data file. 

The consumer 2 60 in this case is a video playback 
system which produces a television program in real-time. 

15 This can be a standard media player program running on a 
computer. The data stream 2 55 would consist of audio 
samples and video frames corresponding to the television 
program. In the simplest embodiment of this application, 
the control stream 256 consists of two tokens, start and 

2 0 end, corresponding to the start and end of a playback. 

In this case the index file 2 85 will contain only two 

event-data pointer records: (start,l,TQ) and (end ^ N ^T>^2) ' Here 
E^^^^'' Start'' and £2,2="^^*^"- ^2.1 N is the total number of 
frames in the video sequence and j ~Tq is the time at 

2 5 which the start of playback was logged. 

If the playback occurs at the standard NTSC video rate 

of 3 0 frames per second, then given the starting time Tq 
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for the playback, it is possible to calculate, for each 
subsequent time T , the data pointer V for the video frame 
which appeared at that instant in time. It is given by 

V{T)-\?>^{T-Tq)\, where time is measured in seconds and V 

5 is an integer corresponding to a sequential video frame 
number. denotes the floor function (x is a placeholder 

value) for a real number x which returns the largest 
integer that is less than or equal to x . For example, if 

X = 3,5, then [xj - 3, We call V a ^'virtual data pointer'' 

10 since it does not appear in the index file directly but can 
be calculated from the contents of the index file. 

Once index files 235 and 2 85 have been constructed for 
a particular television program, playback with closed 
caption information can be accomplished. Given the data 

15 pointer V for a video frame which is to be produced, the 
corresponding start and end times for that frame can be 
obtained by inverting the function V{T) given above. By 

searching for these times in index file 235, the 
appropriate closed caption text can be identified and 
20 displayed. 

In many cases it is difficult for a typist to keep up 
with the dialog in a television program or movie in real- 
time. The dialog can be very fast and may be difficult to 
understand. It may be desirable to allow the typist to 
25 control the playback rate for the application, slowing down 
or speeding up the rate at which frames are produced. This 
varies depending on the difficulty of the transcription 
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task, but is easily accomplished based the techniques of 
the present invention. 

In order to accommodate a variable playback rate, we 
must expand the control stream 256 to include the desired 
5 playback rate as part of the control information. The event 
detector 265 generates an event whenever the playback rate 
changes. Event-data pointer records contained in index 285 

will then have the form {Rj.Vj.T^j) / where Rj is the 

playback rate in effect following video frame Vj . 

10 The calculation of virtual data pointers can be 

amended in a similar manner. For a particular time we 
can identify the ktia event-data pointer record in the index 
285 which has the property that T^^^<T <T^^j^^^ . We then have 



15 flexibility of the present invention in dealing with data 
streams whose data rate varies in an unpredictable fashion. 

The calculation of a virtual data pointer in the 
variable playback case also illustrates a potential 
advantage of the present invention over the prior art. For 

2 0 example, by including data pointers directly in the event 
records, we gain robustness against system latency. 

Latency refers to the delay between the issuance of an 
event -data pointer pair by the data pointer generator and 
time it is processing by the event-data pointer logger 250. 

25 An event can occur at a particular time t^, but the event 

may not be logged until time Tj==^tj+ATj, where ATj is the 




This example illustrates the 
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latency. In this case the time-stamp in the index record 
152 will not accurately reflect the time at which an event 
occurred. 

There are many possible sources of latency. If the 
5 logger unit 250 receives a large number of event -data 

pointer pairs, it may not be able to process them quickly 
enough, particularly if the computer system is running the 
application is experiencing a large workload. If the pairs 
are being transmitted over a network, then congestion in 

10 the network is another potential source of latency. The 

consequences of latency depend upon the application and the 
desired time resolution at which queries are being made. 
However, a general goal is to minimize latency. 

Consider a case where data pointers are not recorded 

15 for each event corresponding to a change in the playback 
rate. This is typically the case in straightforward event - 
logging systems such as SYSLOGD, In that case it is still 
possible to calculate a virtual data pointer. Consider the 
specific case where is 0 and calculate virtual data 

20 pointers for time instants from T-^ to T, where T^<T<T^. 

Note that the stream index has been dropped for simplicity, 
i.e., Ts^j is now Tj . Then we have 

V^(T) = R,(T,-T,) + R,(T,-T,) + R,{T-T,), where V^iT) is the 

virtual data pointer calculated using event information 
25 alone. Note that for simplicity the floor function has been 
omitted from the above and following computations. 
Substituting in the latencies, we can compute the error in 
the virtual data pointer in the event case : 
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- R, (AT^ - at; ) + (AT; - at; ) - R,AT, 



where V^(T) denotes the correct value for the data pointer 

in the absence of any latency in the system. Similarly we 
can compute the error in the virtual data pointer generated 
5 by the current invention: 

ViT) = V{T) - KiT) = V, + R,{T-t,- AT,)- V^iT) = -R,AT, . 

Note how the error in this case is only dependent on the 
most recent latency, aT;^^ whereas the error in the virtual 
data pointer for the prior art is affected by the history 

10 of latency errors, aT^, aT^, and aT^^ This illustrates the 
benefit of the present invention over other systems that 
record events but not associated data pointers. 

An alternative solution would be to eliminate latency 
altogether by time-stamping events before they are 

15 transmitted to the logger rather than after they arrive. 
This approach has a significant disadvantage, however, to 
the extent that it would require all of the event detection 
modules to have synchronized clocks. This could be 
difficult to achieve if the modules are executing on a 

20 distributed computer system. 

Fig. 5 is yet another exemplary recording system for 
closed caption generation according to the principles of 
the present invention. 

Audio device 62 0 generates audio output at speaker 670 

25 that is heard by user at keyboard 610. As sound such as 
words are played at speaker 670, corresponding text is 
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generated at keyboard 610 and supplied to text entry system 
63 0 such as a word processor. 

In addition to driving speaker 670, the output of 
audio device 62 0 is fed into a speech recognition system 
5 660. Audio playback rate controller 650 compares the 
present average text input rate of keyboard 610 with the 
present average speech output rate of speaker 670. If the 
average text input rate is faster than the average rate of 
words played by speaker 670, the audio playback rate 

10 controller increases the playback speed of the audio 

device. Conversely, the playback rate of the audio device 
is reduced when the audio playback rate controller senses 
that the average text input rate is slower that the average 
rate of words played by the speaker 670. In this way, the 

15 audio playback rate controller 650 automatically adjusts 
the playback rate to the present typing rate of user at 
keyboard 610. This alleviates the need for a transcriber 
to manually adjust the speed of an audio file as 
corresponding text is generated. 

20 Data link 640 such as data stream correlator system as 

shown in Fig. 1 provides linking of data streams based on 
time-stamps generated by a common system clock. Audio 
device 62 0 generates an audio data stream that is 
correlated with a text-based data stream as previously 

2 5 described. In one embodiment, an event in audio data 

stream is optionally based upon a predetermined passage of 
time. Alternatively, events within audio data stream are 
defined by a passage of a predetermined amount of tape 
(analog recording) or a fixed number of audio 

30 samples (digital recording). 
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The principles of the present invention have 
applications beyond note taking, text entry, or a close 
captioning system. One example is a video surveillance 
system for an automobile parking lot. This example system 
5 uses multiple video cameras. One set of cameras record the 
automobiles and associated license plates at the entrance 
and exits. Another set of cameras record the parking area. 
In current video surveillance systems, cameras continuously 
record video on tape. This results in a waste of video 

10 recording media. A more efficient system would record, for 
example, only for a few seconds after a motion detector 
indicates when a car has entered or left the garage or 
parking space. This type of sporadic recording renders it 
difficult to determine where an event is recorded on a 

15 videotape. Even a time-stamped videotape must be searched 
for the appropriate time match. 

The features of the present invention can be used to 
efficiently link the data from multiple video recorders. 
The events can be the start of video recording as triggered 

2 0 by the motion detectors and the data pointers would be the 

physical location (for example, a tape counter value) where 
the video recording starts. Our system links the video 
recording of cars and license plates entering and leaving 
the lot and the video recording of parking activity within 
25 the lot. For example, if a car hits another parked car and 
leaves without reporting the incident, the video recording 
of the parking lot activity can be quickly reviewed for all 
car activity next to the damaged car. Once suspect cars are 
identified in the parking lot video, time-stamped events 

3 0 and data pointers can be reviewed to link the video of 
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these cars to the video recorded at the lot entrance and 
exits, so the car's license plates can be identified. 

While this invention has been particularly shown and 
described with references to preferred embodiments thereof, 
5 it will be understood by those skilled in the art that 
various changes in form and details may be made therein 
without departing from the scope of the invention 
encompassed by the appended claims. 
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CLAIMS 

What is claimed is: 



11. A method of correlating time sequenced data streams 

2 comprising: 

3 identifying events within a first data stream; 

4 generating positional information, also known as 

5 data pointers, for identified events; 

6 assigning a time- stamp to each event; and 

7 correlating data from the first data stream to 

8 data in a second data stream based on the time-stamped 

9 data pointers. 

12. A method as described in claim 1, wherein the first 

2 data stream is a word processing data stream and the 

3 second data stream is an audio data stream. 

13. A method as described in claim 1, wherein the time- 
2 stamp is generated by a common system logger. 

14. A method as described in claim 1, wherein the system 
2 logger time stamps data in the second data stream. 

15. A method of correlating events in data streams 

2 comprising the steps of: 

3 detecting events within a first data stream and 

4 assigning corresponding data pointers to the events; 
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5 detecting events within a second data stream and 

6 assigning corresponding data pointers to the events; 

7 and 

8 utilizing the data pointers to link events in the 

9 first data stream to events in the second data stream. 

16. A method as described in claim 5 further comprising 

2 the step of : 

3 in addition to assigning data pointers, assigning 

4 time-stamps to the events in the first data stream and 

5 the second data stream to correlate events in the 

6 first data stream with events in the second data 

7 stream. 

17. A method as described in claim 6 further comprising 

2 the steps of: 

3 identifying an event in the first data stream; 

4 and 

5 locating of a corresponding event in the second 

6 data stream using the time-stamps as an index. 

18. A method as described in claim 6, wherein the time- 

2 stamps assigned to the first data stream and second 

3 data stream are generated from a common system clock. 

19. A method as described in claim 6, wherein the time- 

2 stamps assigned to the first data stream and second 

3 data stream are generated from separate but 

4 synchronized clocks. 
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1 10. A method as described in claim 5, wherein the first 

2 data stream is asynchronous. 

1 11. A method as described in claim 5 further comprising 

2 the step of: 

3 presenting the first data stream with a time- 

4 varying playback rate. 

1 12. A method as described in claim 5, wherein an event in 

2 the first data stream is defined by a fixed passage of 

3 t ime . 

1 13 . A method as described in claim 5, wherein the first 

2 data stream includes data generated by a word 

3 processor and the second data stream includes audio 

4 data. 

1 14 . A method as described in claim 5 further comprising 

2 the step of: 

3 varying a rate of generating the first data 

4 stream relative to the second data stream based on 

5 detection of events in the second data stream. 

1 15. A method as described in claim 5, wherein the first 

2 data stream includes audio data and has a playback 

3 rate dependent on detection of corresponding textual 

4 data in the second data stream. 

1 16. A method as described in claim 5, wherein the events 

2 in the first stream are random events. 
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1 17. A method as described in claim 5 further comprising 

2 additional data streams in which events are 

3 correlated. 

1 18. A method as described in claim 5, wherein the first 

2 data stream and second data stream are recorded to a 

3 storage device for later retrieval and the 

4 corresponding data pointers indicate a location of a 

5 corresponding event recorded in the storage device. 

1 19. An apparatus for correlating time sequenced data 

2 streams comprising: 

3 an event detector for identifying events within a 

4 first data stream and generating positional 

5 information for the identified events; and 

6 a system logger for assigning a time-stamp to 

7 identified events and correlating the time-stamped 

8 data in the first data stream with a second data 

9 stream. 

1 20. An apparatus as described in claim 19, wherein the 

2 first data stream is a word processing data stream and 

3 the second data stream is an audio data stream, 

1 21. An apparatus as described in claim 19, wherein the 

2 time-stamp is generated by a common system logger. 
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1 22. An apparatus as described in claim 19, wherein the 

2 system logger time-stamps data in the second data 

3 stream. 

1 23. An apparatus for correlating events in data streams 

2 comprising: 

3 a first event detector for identifying events 

4 within a first data stream; 

5 a second event detector for identifying events 

6 within a second data stream; and 

7 a system logger that generates data pointers to 

8 correlate an event in the first data stream to an 

9 event in the second data stream. 

1 24. An apparatus as described in claim 23, wherein the 

2 system logger assigns time-stamps to the events in the 

3 first data stream and the second data stream. 

1 25, An apparatus as described in claim 24, wherein the 

2 time -stamps are used as an index to approximate a 

3 location of an event in one data stream to an event in 

4 another data stream. 

1 26, An apparatus as described in claim 24, wherein the 

2 time-stamps assigned to the first data stream and 

3 second data stream are generated from a common system 

4 clock, 

1 27, An apparatus as described in claim 24, wherein the 

2 time-stamps assigned to the first data stream and 
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3 second data stream are generated from separate but 

4 synchronized clocks. 

1 28. An apparatus as described in claim 24, wherein the 

2 first data stream is asynchronous. 

1 29. An apparatus as described in claim 24 further 

2 comprising: 

3 a playback device that generates the first data 

4 stream and varies a rate of a pre-recorded data 

5 stream. 

1 30. An apparatus as described in claim 23, wherein an 

2 event in the first data stream is defined by a fixed 

3 passage of time. 

1 31. An apparatus as described in claim 23, wherein the 

2 first data stream includes data generated by a word 

3 processor and the second data stream includes audio 

4 data. 

1 32. An apparatus as described in claim 23 further 

2 comprising the step of: 

3 varying a rate of the first data stream relative 

4 to the second data stream based on detection of events 

5 in the second data stream. 

1 33. An apparatus as described in claim 23, wherein the 

2 first data stream includes audio data and has a 
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3 playback rate dependent on detection of corresponding 

4 textual data in the second data stream. 



1 34. An apparatus as described in claim 23, wherein the 

2 events in the first stream are random events. 

1 35. An apparatus as described in claim 23, wherein 

2 additional data streams are correlated to the first 

3 data stream. 

1 36. An apparatus as described in claim 23, wherein the 

2 first data stream and second data stream are recorded 

3 to a storage device for later retrieval and the 

4 corresponding data pointers indicate a location of a 

5 corresponding event recorded in the storage device. 
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METHOD AND SYSTEM FOR CORRELATING DATA STREAMS 

ABSTRACT OF THE DISCLOSURE 

Events in a first data stream are correlated with 
5 events in a second data stream based on a method of 

indexing. As information of a first and second data stream 
are stored to a data file, the data streams are monitored 
for certain trigger events. An informational data set is 
generated and stored in an index file for each detected 

10 trigger event. Preferably, each informational data set 
includes the type of detected event, a time-stamp 
indicating when the event occurred and a data pointer 
indicating where the data associated with the event is 
stored in a corresponding data file. When time-stamps of 

15 detected trigger events in both data streams are generated 
based on a common system clock, data in one data stream is 
correlated with data in another data stream based on the 
time-stamps . 



Data sets for events 
in data stream #1 
5 event = el, data ptr = 100, time = tl 

event - e2, data ptr ^ 300, time = t3 

1 0 event = e5, data ptr = 500, time = tlO 

event = e3, data ptr ~ 700, tmie ^ t6 

event = e4, data ptr = 800, time = t8 
1 5 event = e6, data ptr = 1 1 00, time = tl 1 



Data sets for events 
in data stream #2 

event = fi, data ptr = 500, trnie = t2 

event = fl, data ptr - 1000, time = t4 
event = f3, data ptr = 1400, time ^ t5 

event = f4, data ptr = 1700, time = t7 

event = f5, data ptr = 2200, time = t9 



